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= AV safety is complicated 
e Mishaps and other issues 
e US regulatory landscape 


= Dirty Dozen AV industry myths 


= Deployment governance 


e Governance is #1 AV ethics issue now 1 
> ww “itips://on fB rete 





= For 2022, the big issues are regulation & trust 
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= Public expectations 
e Expect super-human machine performance 
e Trust too easily given, backlash when broken 
= Technical challenges 
e Machine Learning safety is work in progress 
e Statistical techniques struggle with rare events | — | 
= Historical industry culture clash 
e Autonomy researchers: it’s all about the cool small-scale demo 
e Silicon Valley: move fast + break things 
e Automotive: blame driver for not mitigating equipment failures 
e Regulators: test-centric; weak digital safety expertise 
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How's It Going With Autonomy Testing? i Mellon 


University 
= Uber ATG fatality, Tempe AZ/US: March 2018 
e Uber ATG closed: January 2021 
= Local Motors injury, Whitby CA: Dec. 2021 
e Company closed: Jan. 2022 ye ORE L 
= Pony.Al crash: CA/US: Oct. 2021 ro oh. mie, 4 ra - ' ¥ 
e Uncrewed test permit revoked bt ee), a Je 
m WeRide sleeping tester: Oct. 2021 = = Se 
e Company deflects issue / no apparent regulator SaBon . 
= Easymile shuttle phantom braking injuries: (2019, 2020) 
= Cruise battling with SF MTA over passenger pickup: Dec 2021 
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= Tesla FSD “beta test” US: multiple incidents 
e Videos of reckless driving by testers 
e Exploiting Level 2 loophole for L3/L4 testing me ‘ Bug 
= Tesla AutoPilot: injuries and fatalities + 
e Multiple crashes investigated by NTSB wz es a 
— Common theme: inadequate driver monitoring | - 
e NHTSA investigations = 
— 11 crashes; 17 injuries; 1 fatality y/ -" 
for crashes into emergency vehicles/workers tips:/oitiy33.08k7 _prtps://bit ly/3naunX 


e NHTSA mandating reporting from all Level 2 and higher vehicles 
e Felony charges for fatal AP-related crash (Jan. 2022) nitps://pit.ty/3tFiqou 
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US Regulatory Posture As Of Early 2022 fin 
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= Federal / equipment safety: almost nothing for AVs 
e NHTSA ANPRM proposing industry standards Dec. 2020 
e Started collecting crash data in 2021 : 

= State / driver safety: administrative only | 
e California: permits, driver checks, reporting 
e Texas, Arizona, etc. “open for business” 
e Aggressive state-by-state lobbying 

= Municipal / local conditions: mixed 3 
e NYC DOT requires SAE J3018 for testing >. 


e Industry pushing for state preemption 
in Pennsylvania 
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: Deployment Governance Ni 
= Who decides it's time to deploy, based on what? 


u US: self-certification to FMVSS (no homologation) 
e Vehicle test of basic functions only 








e No requirement for engineering standards — — 
e State permits are licensing & insurance 
» Regulations based on SAE J3016 Level 
— J30716 is nota safety standard! Issued 9014-01 
° ° Revised 2021-04 
= Companies decide when to test/deploy | ~~ 
SURFACE VEHICLE 
e Opaque about their safety goal RECOMMENDED PRACTICE 
@ Opaque about criteria to deploy (R) Taxonomy and Definitions for Terms Related to 
Driving Automation systems tor On-Road Motor Vehicles 


e Enormous pressure: SBillion milestones i ee ae 
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Eroding Trust: AV Regulatory Dirty Dozen Myths Weller... 


1) 94% fatalities due to human error 
e Humans failed to prevent human caused 


“The critical reason was assigned to drivers in an 
estimated 2,046,000 crashes that comprise 94 percent 


of the NMVCCS crashes at the national level. [DoT HS 812 115] 


However, in none of these cases was the assignment 
intended to blame the driver for causing the crash.” 


2) Innovate or regulate; choose only one 
e False dilemma, especially for testing safety 
e ‘Innovative’ is not necessarily safe 


3) Already sufficient regulations in place 





NHTSA 


Benefits of Automation 

SAFETY 

The safety benefits of automated vehicles are paramount. Automated 
vehicles’ potential to save lives and reduce injuries is rooted in one 
critical and tragic fact: 94 percent Of Serious Crashes are due to human 
@rron Automated vehicles have the potential to remove human error 
from the crash equation, which will help protect drivers and 
passengers, as well as bicyclists and pedestrians. When you consider 
more than 35,092 people died in motor vehicle-related crashes in the 
U.S. in 2015, you begin to grasp the lifesaving benefits of driver 
assistance technologies. 


https://www.nhtsa.gov/technology- 
innovation/automated-vehicles-safety 


e State DOTs require license+tinsurance, not testing safety ; 
© 2022 Philip Koopman 8 


AV Regulatory ity Dozen Myth 525, ee 


University 


4) Cars meet all existing safety regulations 


e FMVSS is about basic features, not software safety 
e FMVSS needs updates, but not main issue 


5) Safety standards aren't applicable 
because {reasons} 


e Example: adopting safety standards 
will force AVs to be less safe 


e Road testing: SAE J3018 + SMS practices 
— Be mindful of lessons learned by Uber ATG 


e Production: ANSI/UL 4600, incorporating ISO 26262, ISO 21448 
— This is what is in NHTSA ANPRM from Dec. 2020 
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Waymo Standard Statement Bele 


= Waymo Safety Methodologies report, Oct 2020 


e 26262 is great stuff! 

e But only looked at HARA (part 3), not whole standard 

e “does not rely’ — doesn't say they use HARA at all! 

e “not a perfect fit’ — Myth #5 

e Summary: “We looked at ISO 26262 HARA, but decided not to.” 
action to address the hazard. ISO 26262 has provided significant insights for Waymo's hazard 
analysis processes. However, Waymo does not rely strictly or exclusively on ISO 26262's 
principles, which are not a perfect fit for a Level 4 ADS, where there is a need for a special 
focus on the plethora of conditions likely to be encountered in the intended ODD, and where 
separate analysis of individual items may not be as useful as analysis of hazards related to 


system interactions. https://bit.ly/3KF3IGJ 
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6) Avoid “patchwork” of state regulations 
e Patchwork primarily of AV industry making 


7) “Spirit of” / “incorporates ideas from” 
language regarding standards 
e If it doesn't conform, it’s just hand-waving 
e Maybe theyre doing the right thing ... 
... but maybe not Wi. 
e If you can’t show it is safe, AS OK 
it should be presumed unsafe 1 SSS 





) 
https://bit.ly/35jdWwAz 
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Aurora VSSA Standard Statement eee 





= Aurora Voluntary Safety Self-Assessment 2021 


e “We leverage industry best practices and standards 
.. relevant principles) — Myth #7 


e Laundry list of standards “are applicable” 


... but no commitment to use them. 
safety within Aurora. We leverage industry best practices and standards that 
have significantly raised the bar in automotive safety, and we are committed 
to ensuring the relevant principles are embedded within our internal policies, 
Orocedures, and approaches in developing the Aurora Driver. 


The US. DOT Comprehensive Plan” identified a set of twenty automotive best 


oractices and standards, which individually address different components or 

processes for a self-driving system or enterprise. Further, standards such as SAE 

J3016, SAE J3018, UL 4600, International Organization for Standardization (ISO) 

26262, and ISO 21448, as well as the best practices from the Automated Vehicle . oy. ; 
ps://bit.ly/SlaajLw 

Safety Consortium (AVSC), are applicable to self-driving technology. © 2022 Philip Koopman 12 
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8) Government regulators aren't smart enough 
e US DOT proposed using industry's own standards 
9) Disclosing data gives away secret sauce SCS 
e How is # of reckless driving incidents secret autonomy sauce? 
10) Delaying AV deployment is basically killing people 
e No data exists showing AVs are actually safer in practice 
11) We haven't killed anyone (yet), so we're safe 
e At 100M miles+ per fatality, insufficient data to prove safety 
12) Other cities/states don’t regulate 
e Intimidation, FOMO for economic benefits, “hostile to innovation” 
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Would you trust your life to an automated vehicle? 





“77 - 
> 


AMA ITER 
OF TRUST 


Ford VSSA 2021 https://bit.ly/3njionT © 2022 Philip Koopman 14 
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Industry Behaviors That Erode Trust Ee 
= Messaging the Dirty Dozen Myths 
e Safety theater of various forms aia! 
= Maintaining option less safe deployment Autonomous 


e Possibly employing Moral Elune Zones Vehicle Myths: 
e AV Secret Sauce excuse for opacity he Dirt 
= Lobbying states for favorable terms the Dirty 
e Preempt municipal ordinances Dozen 
e FOMO-driven narrative (China; other states) © 2 5° ——supsyybiiy/ananoe 
e Sue the ADS; Low insurance limits ($1M vs. $11M statistical life) 
e No defined level of safety, nor gov't oversight over safety 
e Example: PA SB 965 (Jan 5, 2022) https://it.ly/31c6LiE Ms nite. rs 


= Safety transparency 
e Road testing safety metrics 
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SaFAD/ISO 
TR 4804 








e Deployment safety criteria SYSTEM U1 4600 
= Commit to industry standards BrAM sop 

e SAE J3018+SMS road testing safety we 

© ISO 26262 + ISO 21448 + aie 


ANSI/UL 4600 for deployment safety = 


SAE 
21434 


ellon 
University 
Safety Beyond 
Dynamic 
Driving 
Environment & 
HIGHLY 
apace C8 AUTOMATED 
VEHICLE 
Equipment SAFETY 
Faults CASE 
Computer UL 4600 


Security 





SAFETY 


= Collaborate with regulators 


NCAP 


Basic 
Vehicle 
Functions 








e Perhaps one high profile crash away from forcing regulators to act 
e Balance benefit to industry with protection for road users 
e Include all stakeholders to find win-win arrangement 
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= AV industry at a crossroads: 
1. Adversarial to regulation; risk of backlash, or 
2. Collaborative governance to establish trust 






= Should companies own safety governance? 
e Huge financial benefits for early to market 
e S$ Billion funding and milestone pressure 
e Tesla behavior: no consequences; huge stock price gains 
— How long will other companies resist temptation to cut safety corners? 


= Detailed paper on AV Regulation and Trust: 
e http://dx.doi.org/10.2139/ssrn.3969214 (UCLA J. Law & Tech.) 
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